Reserve power system for data center

ABSTRACT

A system for performing computing operations in a data center includes one or more sets of computer systems, one or more primary power systems, and a reserve power system. The primary power systems include at least one power distribution unit that supplies power to at least one of the sets of computer systems. The reserve power system automatically supplies power to at least one of the sets of computer systems if a condition is met (such as a failure of the primary power system).

This application is a Continuation of U.S. application Ser. No.12/825,198, filed Jun. 28, 2010, which is incorporated by referenceherein in its entirety.

BACKGROUND

Organizations such as on-line retailers, Internet service providers,search providers, financial institutions, universities, and othercomputing-intensive organizations often conduct computer operations fromlarge scale computing facilities. Such computing facilities house andaccommodate a large amount of server, network, and computer equipment toprocess, store, and exchange data as needed to carried out anorganization's operations. Typically, a computer room of a computingfacility includes many server racks. Each server rack, in turn, includesmany servers and associated computer equipment.

Because the computer room of a computing facility may contain a largenumber of servers, a large amount of electrical power may be required tooperate the facility. In addition, the electrical power is distributedto a large number of locations spread throughout the computer room(e.g., many racks spaced from one another, and many servers in eachrack). Usually, a facility receives a power feed at a relatively highvoltage. This power feed is stepped down to a lower voltage (e.g.,110V). A network of cabling, bus bars, power connectors, and powerdistribution units, is used to deliver the power at the lower voltage tonumerous specific components in the facility.

Some data centers have no redundancy at the PDU level. Such data centersmay have a large affected zone when a UPS or PDU failure in the powersystem occurs. In addition, some data centers have “single threaded”distribution via the electrical supply to the floor, and in whichmaintenance can only be performed when the components are shut-off.

Some data centers include back-up components and systems to provideback-up power to servers in the event of a failure of components orsystems in a primary power system. In some data centers, each primarypower system may have its own back-up system that is fully redundant atall levels of the power system. For example, in a data center havingmultiple server rooms, each server room may have its own primary powersystem and back-up power system. The back-up system for each server roommay have a switchboard, uninterruptible power supply (UPS), and floorpower distribution unit (PDU) that mirrors a corresponding switchboard,uninterruptible power supply, and floor power distribution unit in theprimary power system for that server room. Providing full redundancy ofthe primary power systems may, however, be very cost both in terms ofcapital costs (in that in may require a large number of expensiveswitchboard, UPSs, and PDUs, for example) and in terms of costs ofoperation and maintenance. In addition, with respect to the primarycomputer systems, special procedures may be required to switchcomponents from the primary system to a back-up system, furtherincreasing maintenance costs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating one embodiment of a data centerhaving a reserve power system that backs up primary power systems formultiple rooms of a data center.

FIG. 2 is a block diagram illustrating an embodiment of a reserve powersystem that includes automatic transfer switches at a UPS level and a atswitchboard level.

FIG. 3 is a schematic illustrating one embodiment of a reserve powersystem having power distribution units for rack systems in multiple podsof a data center.

FIG. 4 is a schematic illustrating one embodiment of transformation anddistribution sections of a reserve power system.

FIG. 5 is a schematic diagram illustrating an arrangement of a rackincluding an automatic transfer switch in a data center.

FIG. 6 illustrates a reserve power system with a PDU that providesreserve power for computer systems on multiple primary PDUs.

FIG. 7 illustrates providing reserve power for computer systems in adata center.

FIG. 8 is a flow diagram illustrating failover logic for a reserve powersystem.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription thereto are not intended to limit the invention to theparticular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and

alternatives falling within the spirit and scope of the presentinvention as defined by the appended claims. The headings used hereinare for organizational purposes only and are not meant to be used tolimit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include,” “including,” and“includes” mean including, but not limited to.

DETAILED DESCRIPTION OF EMBODIMENTS

Various embodiments of a reserve power system for computer systems in adata center are disclosed. According to one embodiment, a system forperforming computing operations in a data center includes two or moresets of computer systems, one or more primary power systems, and areserve power system. Each set of computer systems may correspond, forexample, to a rack in the data center. In certain embodiments, thereserve power system is “oversubscribed” such that the total powerrequirements of computer systems coupled to the reserve power systemexceed the capacity of the reserve power system to supply power at anygiven time. Thus, the reserve power system may not be fully redundant tothe primary power system. The reserve power system can automaticallyswitch one or more of the sets of computer systems from primary power toreserve power independently of other sets of computer systems such thatonly some of the sets of computer systems are on reserve power whileothers of the sets of the computer systems remain on primary power.

According to one embodiment, a system for providing reserve power tocomputer systems in a data center including a reserve power system. Thereserve power system automatically supplies power to at least two setsof computer systems if a condition is met (such as a failure of aprimary power system. The reserve power system includes two or moreautomatic transfer switches. Each of the automatic transfer switches mayprovide automatic switching to reserve power for a different set ofcomputer systems

According to one embodiment, a method of supplying reserve power tocomputer systems in a data center includes coupling a reserve powersystem to two or more sets of computer systems. The computer systemsreceive power from a primary power system. In certain embodiments, thereserve power system is “oversubscribed” in that the total powerrequirements of computer systems coupled to the reserve power systemexceed the capacity of the reserve power system. One or more of the setsof computer systems can be automatically switched from primary power toreserve power (for example, upon failure of primary power to the sets ofcomputer systems), while primary power is maintained to other of thesets of computer systems.

In various embodiments, redundant power is provided for many differentpower distribution systems. In one embodiment, power redundancy is sizedsuch that a system can support any N distribution system failures. Insome embodiments, a reserve power system is oversubscribed to achieveN+1 redundancy for a data center (rather than, for example, 2N powerredundancy). In some embodiments, a system having less than one-to-oneredundancy may include overload protection, such as a breaker system, toprotect against overload of a reserve power system.

In some embodiments, a reserve power system provides back up power forsystems and components from top to bottom in a power distribution chain.In certain embodiments, a reserve power system backs up a primary powersystem including a transformer that receives power from a utility feed,a backup generator for the utility feed, a switchboard that receivespower from the transformer, one or more UPSs that receive power from theswitchboard, and one or more power distribution units.

As used herein, “computer room” means a room of a building in whichcomputer systems, such as rack-mounted servers, are operated.

As used herein, “data center” includes any facility or portion of afacility in which computer operations are carried out. A data center mayinclude servers dedicated to specific functions or serving multiplefunctions. Examples of computer operations include informationprocessing, communications, simulations, and operational control.

As used herein, “operating power” means power that can be used by one ormore computer system components. Operating power may be stepped down ina power distribution unit or in elements downstream from the powerdistribution units. For example, a server power supply may step downoperating power voltages (and rectify alternating current to directcurrent).

As used herein, “wye transformer” or “Y transformer” refers to atransformer whose secondary windings are in a wye, or Y, configuration.

As used herein, “delta transformer” refers to a transformer whosesecondary windings are in a delta configuration.

As used herein, “power distribution unit” means any device, module,component, or combination thereof, that can be used to distributeelectrical power. The elements of a power distribution unit may beembodied within a single component or assembly (such as a transformerand a rack power distribution unit housed in a common enclosure), or maybe distributed among two or more components or assemblies (such as atransformer and a rack power distribution unit each housed in separateenclosure, and associated cables, etc.). A power distribution unit mayinclude a transformer, power monitoring, fault detection, isolation.

As used herein, “primary power” means any power that can be supplied toan electrical load, for example, during normal operating conditions.

As used herein, “floor power distribution unit” refers to a powerdistribution unit that can distribute electrical power to variouscomponents in a computer room. In certain embodiments, a powerdistribution unit includes a k-rated transformer. A power distributionunit may be housed in an enclosure, such as a cabinet.

As used herein, “rack power distribution unit” refers to a powerdistribution unit that can be used to distribute electrical power tovarious components in a rack. A rack power distribution may includevarious components and elements, including wiring, bus bars, connectors,and circuit breakers.

As used herein, “remote power panel” means any panel, device, module,component, or combination thereof, that can be used to transfer ordistribute electrical power from one or more input conductors to one ormore output conductors. In certain embodiments, a remote power panelincludes main lug only panel conductors. A remote power panel may behoused in an enclosure, such as a cabinet.

As used herein, “reserve power” means power that can be supplied to anelectrical load upon the failure of, or as a substitute for, primarypower to the load.

As used herein, “source power” includes power from any source, includingbut not limited to power received from a utility feed. In certainembodiments, “source power” may be received from the output of anothertransformer (which is sometimes referred to herein as “intermediatepower”).

As used herein, “computer system” includes any of various computersystems or components thereof. One example of a computer system is arack-mounted server. As used herein, the term computer is not limited tojust those integrated circuits referred to in the art as a computer, butbroadly refers to a processor, a server, a microcontroller, amicrocomputer, a programmable logic controller (PLC), an applicationspecific integrated circuit, and other programmable circuits, and theseterms are used interchangeably herein. In the various embodiments,memory may include, but is not limited to, a computer-readable medium,such as a random access memory (RAM). Alternatively, a compact disc—readonly memory (CD-ROM), a magneto-optical disk (MOD), and/or a digitalversatile disc (DVD) may also be used. Also, additional input channelsmay include computer peripherals associated with an operator interfacesuch as a mouse and a keyboard. Alternatively, other computerperipherals may also be used that may include, for example, a scanner.Furthermore, in the some embodiments, additional output channels mayinclude an operator interface monitor and/or a printer.

FIG. 1 is a block diagram illustrating one embodiment of a data centerhaving a reserve power system that backs up primary power systems formultiple rooms of a data center. Data center 50 includes rack 52,primary power side 100, and reserve power side 200. Primary power side100 includes transformer 102, generators 104, and switchgear 105, andprimary power systems 106. Sets of computer systems 54 in racks 52 mayperform computing operations in data center 50. Computer systems 54 maybe, for example, servers in a server room of data center 50. Computersystems 54 in racks 52 may each receive power from one of primary powersystems 106. In one embodiment, each of primary power systems 106corresponds to, and provides power to, the servers in one room in datacenter 50. In one embodiment, each of primary power systems 106corresponds to, and provides power to, one rack system in data center102.

Primary power systems 106 each include UPS 110 and floor powerdistribution unit 112. Floor power distribution unit 112 provides powerto various racks 52. In some embodiments, floor power distribution unit112 includes a transformer that transforms the voltage from switchgear105. Each of rack 52 may include a rack power distribution unit 56. Rackpower distribution units 56 may distribute power to computer systems 54.

Transformer 102 is coupled to a utility feed. The utility feed may be amedium voltage feed. In certain embodiments, the utility feed is at avoltage of about 13.5 kilovolts or 12.8 kilovolts at a frequency ofabout 60 Hz. Generators 104 may provide power to primary power systems106 in the event of a failure of utility power to transformer 102. Inone embodiment, one of generators 104 provides back-up power for each ofprimary power systems 106. UPS 110 may provide uninterrupted power toracks 52 in the event of a power failure upstream from UPS 110.

Reserve power system 200 may provide reserve power for all of thecomputer systems 54 supplied by primary power systems 106. In someembodiments, reserve power system 200 is powered up at all times duringoperation of data center 50. Reserve power system 200 may be passiveuntil a failure of one or more components of primary power side 100, atwhich time reserve power system 200 may become active.

For illustrative purposes, three primary power systems are shown in FIG.1 (for clarity, details of only the front primary power system 106 areshown). The number of primary power systems 106 on primary power side100 may vary, however. In certain embodiments, a primary power side mayinclude only one primary power system. In addition, the number of powerdistribution units, UPSs, switchgear apparatus may vary from embodimentto embodiment (and, within a given embodiment, from system to system).In some embodiments, primary power system 106 includes many floor powerdistribution units 112. As another example, a primary power system mayhave one UPS that can supply power to many floor power distributionunits.

Reserve power system 202 includes transformer 204 and generator 206.Transformer 204 may supply power to switchgear 208. Critical reservedistribution board 210 may receive power from switchgear 208. Power fromswitchgear 208 may pass through UPS 212. Static switch 214 is providedbetween UPS 212 and critical reserve distribution switchboard 210.Static switch 214 may provide for bypass of UPS 212 (for example, duringmaintenance of UPS 212).

Reserve power system 202 also includes transformer 218 and remote powerpanel 220. Transformer 218 may transform power from critical reservedistribution switchboard 208 and supply power to remote power panels220. Remote power panels 220 may distribute power to servers 54 in racks52. In one embodiment, each of remote power panels 220 of reserve powersystem 202 corresponds to one of floor power distribution units 112 ofone of primary power systems 112. For example, if a floor powerdistribution unit distributes primary power to all of the computersystems in a rack, a remote power panel may distribute reserve power toall of the computer systems in that rack.

Reserve power system 202 also includes an array of automatic transferswitches 222. Automatic transfer switches 222 may control switching ofpower to computer systems 54 between primary power side 100 and reservepower side 200. automatically switch power from one of primary powersystems 106 to reserve power system 202. In some embodiments, oneautomatic transfer switch is provided for each rack system in a computerroom. Thus, an automatic transfer switch may switch input power to therack between one of floor distribution units 112 and one of remote powerpanels 220. In another embodiment, an automatic transfer switch providedfor each half of a rack system. In still another embodiment, automatictransfer switches may be provided at the server level. In certainembodiments, a reserve power system includes manual transfer switches.Manual transfer switches may be used, for example, to enable maintenanceoperations to be performed.

Although in the embodiment shown in FIG. 1, power to servers is switchedbetween primary power and reserve power, in some embodiments, a datacenter may not have automatic transfer switches to switch betweenprimary power and reserve power. In some embodiments, for example,servers in a rack system (such as servers 54 in racks 52) may bedual-supplied by two power systems or include power supplies that accepttwo our more power source inputs. A server may sourced from two powerfeeds without an automatic transfer switch. In some embodiments, aredundant power system for servers in a data center may operate in anactive-active failover configuration. In other embodiments, a redundantpower system for servers in a data center may operate in anactive-passive failover configuration.

Reserve power system 202 further includes controller 230. Controller 230may serve various control functions in reserve power system 202. In someembodiments, controller 230 may control some or all of automatictransfer switches 222 in reserve power system 202. Controller 230includes reserve overload protect circuit 232. In certain embodiments,controller 230 includes at least one programmable logic controller. Theprogrammable logic controller may control some or all of the switchingin or among devices in reserve power system 202.

Shunt trips 236 are provided for each remote power panel 238. Shunttrips 236 may provide overload protection for reserve power system 202.For example, if automatic transfer switches 222 switch too much of theload from computer systems 54 to reserve power system 202, some of shunttrips 236 may shed their respective remote power panels 220 (and thusshut down the computer systems 54 that are receiving power from thoseremote power panels 220). The shedding of computer systems may be based,for example, on priority of the various computer systems receiving powerfrom reserve power system 202. In certain embodiments, shunt trips 236are controlled by overload protect circuit 232 of controller 230.

In some embodiments, each automatic transfer switch is internallycontrolled.

The automatic transfer switch may include fault detection circuitry suchthat when a fault condition is detected in the primary power input, theautomatic transfer switch automatically switches to reserve power. Thus,for the computer systems coupled to the switch, in the event of afailure in any of the elements on primary power side 100 upstream froman automatic transfer switch 222, including floor power distributionunit 112, UPS 110, or switchgear 105, the automatic transfer switch maytransfer input power from primary power to reserve power. Following suchtransfer, the computer systems that have been switched to reserve powermay receive power from remote power panel 220 of reserve power system202. In addition, the computer systems that have been switched toreserve power may be protected against power interruption by UPS 212. Inone embodiment, failover from primary power to reserve power is carriedout within about 8 to about 20 milliseconds.

In some embodiments, a reserve power system is oversubscribed. As usedherein, “oversubscribed” refers to a condition in which total powerrequirements of the systems coupled to a reserve power system exceed thecapacity of the reserve power system (which includes, for example,exceeding the capacity of a sub-system such as a reserve UPS). Forexample, a reserve power system might have 5 rack systems coupled to it,but only be able to provide reserve power to one of the rack systems atany given time. In some embodiments, a reserve power system may beheavily oversubscribed (for example, subscribed at several times thecapacity of the reserve power system). In certain embodiments,oversubscription is applied at a facility-wide level.

In one illustrative embodiment, the total power requirements of computersystems 54 exceed the capacity of reserve power system 200, such thatreserve power system 200 is oversubscribed relative to its capacity. Forexample, the total power requirements of all computer systems 54 in datacenter may exceed 200 KVA, while the load capacity of reserve powersystem 200 may be about 20 KVA. Thus, if all of primary power systems106 failed simultaneously and automatic transfer switches 222transferred all of computer systems 54 in data center 50 to reservepower system 206, reserve power system 206 would not be able to supplypower to all of the computer systems. As is described further herein, areserve power system may, in some embodiments, include overloadprotection against overload caused, for example, by switching of loadsto the reserve power system in excess of capacity. In one embodiment, areserve power system may have multiple computer rooms coupled to thereserve power system, but have the capacity to support one the computersystems of only one computer room at any given time.

In some embodiments, a reserve power system may include automatictransfer between primary power system and a reserve power system atmultiple levels in a power chain. For example, a reserve power systemmay include automatic transfer switches at a UPS level and/or automatictransfer switches at a switchgear level of the power distribution chain.FIG. 2 illustrates an embodiment that includes automatic transferswitches at a UPS level and a switchboard level. Data center 60 includesprimary power side 100 and reserve power side 200. Reserve power side200 includes reserve power system 240. The elements of primary powerside 100 and reserve power side 200 may be generally as described abovewith respect to FIG. 1. Reserve power system 204 shown in FIG. 2,however, includes automatic transfer switches 242 and 244. Automatictransfer switch 242 may automatically switch power between switchgear105 of primary power side and switchgear 208 of reserve power side 200(for example, in the event of a failure of switchgear 105). Automatictransfer switch 244 may automatically switch power between one of UPSs110 of primary power side 100 and UPS 212 of reserve power side 200 (forexample, in the event of a failure of one of UPSs 110).

In some embodiments, a reserve power system protects a switchboard thatprovides power to multiple UPSs. In certain embodiments, a singleswitchboard may provide power to all of the primary power UPSs in a datacenter. In certain embodiments, one or more primary power UPSs and oneor more reserve power UPSs receive power from the same switchboard. Incertain embodiments, a single switchboard may provide power to all ofthe primary power UPSs and reserve power systems in a data center. Invarious embodiments, a reserve power system may back up any or all ofthe UPSs in a data center.

Providing a single reserve transformer and remote power panels, such asis illustrated in FIGS. 1 and 2, may eliminate the need for floor powerdistribution units in a reserve power system. In addition, the costsassociated with a transformer/remote power panel combination may be lessthan that of a floor power distribution unit of equivalent loadcapacity. Nevertheless, in various embodiments, a reserve power systemmay include floor power distribution units (including, for example, aK-rated transformer) instead of, or in addition to, a stand-alonetransformer and remote power panels.

FIG. 3 is a schematic illustrating one embodiment of a reserve powersystem having power distribution units for rack systems in multiple podsof a data center. Data center 300 includes pods 302 and reserve powersystem 304. Each of pods 302 may be a separate physical unit thatprovides computing capacity to data center 300. Each of pods 302 mayinclude rack systems 306 (for clarity, racks systems 306 are shown foronly one of pods 302). Rack systems 306 include computer systems 308.The power load for a rack system may be any amount. In variousembodiments, the power load for a rack system may be 5 KVA, 10 KVA, or15 KVA.

Reserve power system 304 includes reserve PDUs 310. Reserve PDUs 310 mayprovide reserve power to computer systems 308. In one embodiment, eachof reserve

PDUs 310 is rated for 450 KVA. In some embodiments, rack systems 306include rack power distribution units that provide power to computersystems 306 in two-phases, 30 amps per phase.

Reserve power system 304 includes transformer 311, generator 312,reserve bypass switchboard 314, UPS 316, and critical reservedistribution switchboard 318.

Critical reserve distribution switchboard 310 includes automatictransfer switches 320 and automatic transfer switch 322. In someembodiments, automatic transfer switches 320 provide automatic switchingfrom between reserve power and primary power for one or more of sets ofcomputer systems 306 in racks 304. In one embodiment, automatic transferswitches 320 are rated at 700 amps.

Automatic transfer switch 322 may provide for bypassing UPS 316 (forexample, during maintenance of UPS 316). Redundant bus static switch 324may also allow bypass of UPS 316.

Reserve power system 304 also includes shunt trips 326. Shunt trips 326may provide overcurrent protection for reserve power system 304, such asdescribed above with respect to FIG. 1. In one embodiment, shunt trips326 are rated at 700 A.

Transformer 311 may transform power from a utility feed. In oneembodiment, transformer has an output of 480Y/277 volts, three-phasepower. Reserve bypass switchboard 314 includes automatic transfer switch329. Automatic transfer switch 329 may provide automatic switching fromutility feed to generator 312 (for example, in the event of a failure ofutility power. In one embodiment, generator 312 has an output of480Y/277 volts three phase power. In certain embodiments, generator 312is controlled by a PLC.

Although the power output in the example given above is at 480Y/277volts, power voltage in primary power systems and reserve power systemmay in various embodiments be at other voltage levels. Power voltagelevels in may vary, for example, based on the location of a data center.For example, power voltage levels may vary from one geographic region toanother, or from one country to another. In one embodiment, a voltagelevel in a power system (for example, at a transformer output) is about400 volts. In another embodiment, a voltage level (for example, at atransformer output) in a power system is about 600 volts.

FIG. 4 is a schematic illustrating one embodiment of transformation anddistribution sections of a reserve power system. Reserve power system330 includes critical reserve distribution switchboard 332, transformer334, power panels 336, and automatic transfer switches 338. Reservepower from a utility feed (with or without step-down transformation), agenerator, or a UPS may be provided to critical reserve distributionswitchboard 332. In one embodiment, a reserve power system UPS,step-down transformer, and generator are similar to those describedabove with respect to FIG. 3. Thus, in one embodiment, critical reservedistribution switchboard 332 transmits power at 480Y/277 volts,three-phase power, such as from transformer 311 described above withrespect to FIG. 1.

In one embodiment, transformer 334 transforms power to be distributed torack systems 54. Transformer 334 may transform power to a power outputof 480/360-208 volts, three-phase power. In some embodiments,transformer 334 is derated. In certain embodiments, transformer 334derated is based on a K-13 load profile. In one embodiment, transformer334 is derated to 80%.

Power panels 336 distribute reserve power to computer systems in racksystems 52. Each of power panels 336 may provide power to a differentset of computer systems. For example, each power panel may distributepower to one rack or one half-rack (for clarity, only one rack system 52is shown in FIG. 4). In one embodiment, power panels 336 include mainlug only (MLO) conductors. In some embodiments, power panels 336 mayinclude breaker panels. In one embodiment, power panels 336 candistribute up to 600 amps. Although in the embodiment shown in FIG. 4,reserve power system includes 32 power panels in two pods forillustration purposes, a reserve power system may have any number ofpower panels that provide reserve power to any number racks, in anynumber of computer rooms, modules, or pods.

Automatic transfer switches 338 may automatically switch power betweenprimary power and reserve power from reserve power system 330. Primarypower is

transmitted over primary power lines 346. Each of automatic transferswitches 338 may switch power to computer systems in one half rack ofrack 52.

In an embodiment, an automatic transfer switch includes a dual inputincluding a primary and secondary input each at 200-240VAC 24 A, singlephase, 50/60 Hz, and an output at 200-240VAC 24 A, single phase, 50/60Hz. Autoswitching between the primary input and the secondary input maybe based on drop-out of 182 volts, pull-in of 195 volts, based on aninput voltage of 208 V.

Power from transformer 334 may be transmitted on power bus 350. Shunttrip breakers 352 are provided for each of rack systems 52 betweentransformer 334 and power panels 336. In some embodiments, overcurrentprotection for reserve power system 330 is controlled with controller340. Controller 340 may receive data from meter 342 coupled to criticalreserve distribution switchboard 332. In one embodiment, if the load ata reserve UPS (such as UPS 316 described above relative to FIG. 3)exceeds a first predetermined level, and reserve power system 330 tripsinto static bypass, and if the load at the UPS exceeds a secondpredetermined level, some PDUs are shed by way of shunt trip breakers352. In one embodiment, in a system including a 750 KW UPS, a reservepower system trips into static bypass if the load at the UPS is over 700KW, and the lowest priority PDUs are shed if the load at the UPS is morethan 90% of the rating of the static bypass.

FIG. 5 is a schematic diagram illustrating an arrangement of a rackincluding an automatic transfer switch in a data center. Data center 360includes floor 362 and raised floor 364. Primary power source conductors366 and reserve power source conductors 368 are carried on or near floor362. Primary power source conductors 366 and reserve power sourceconductors 368 may terminate at power receptacles 370.

Rack 380 includes automatic transfer switch 382 and power strips 384. Insome embodiments, automatic transfer switch 382 is housed in a separateenclosure. The separate enclosure for the automatic transfer switch maybe rack-mountable. Input cables 386 may plug into power receptacles 370and carry power from primary power source conductors 366 and reservepower source conductors 368 to automatic transfer switch 382. Outputcable 388 may carry power from automatic transfer switch 382 to powerstrips 384. Computer systems in rack 380 may plug into power strips 384and receive power through power strips 384.

FIG. 6 illustrates a reserve power system with a PDU that providesreserve power for computer systems on multiple primary PDUs. Data center400 includes primary power side 402 and reserve power side 404. Primarypower side 402 and reserve power side 404 provide power to loads 406.Loads 406 may be, for example, servers in a data center. Automatictransfer switches may switch power between primary and reserve forindividual loads 406.

Primary power side 402 includes transformer 408, generator set 410, andswitchgear 412. Switchgear 412 may provide power to PDU1-1 431, PDU1-2432, and PDU1-3 433 through UPS1 421. Switchgear 412 may provide powerto PDU2-1 441, PDU2-2 442, and PDU2-3 443 through UPS2 422. Switchgear412 may provide power to PDU3-1 451, PDU3-2 452, and PDU3-3 453 throughUPS3 423.

Reserve power side 404 includes transformer 460, generator set 462, andswitchgear 464. Switchgear 464 may provide power to PDU1-reserve 471,PDU2-reserve 472, and PDU3-reserve 473 through UPS-reserve 466.UPS-reserve 466 may back up any or all of the UPSs on primary side 402.PDU1-reserve 471, PDU2-reserve 472, and PDU3-reserve 473 may back up anyor all of the PDUs on primary side 402.

In some embodiments, each PDU in a reserve power system may providereserve power for computer systems on different primary PDUs. Forexample, PDU1-reserve 471 may provide reserve power for computer systemson PDU1-1 431, PDU1-2 432, and PDU1-3 433; PDU2-reserve 472 may providereserve power for computer systems on PDU2-1 441, PDU2-2 442, and PDU2-3443; and so on. Although in the embodiment shown in FIG. 7, each reservePDU backs up three primary PDUs, a reserve PDU may back up any number ofprimary PDUs.

The embodiment illustrated in FIG. 6 may not include any static transferswitches on the UPS output. This may eliminate the need for the use of asystem control cabinet, and may reduce the “blast radius” of a powersystem failure. The UPS stacks may behave independent of each other andallow resiliency at a smaller component level (for example, a PDU or 25rack blast zone). In some embodiments, the blast zone may be furthermitigated by the installation of dual automatic transfer switches at therack level.

FIG. 7 illustrates providing reserve power for computer systems in adata center. At 480, a reserve power system is coupled to two or moresets of computer systems. Each set of computer systems may be, forexample, the computer systems in a rack system of the data center. Thereserve power system may be oversubscribed such that total powerrequirements of the sets of computer systems exceed the capacity of thereserve power system (including a sub-system or component thereof, suchas a reserve UPS). In certain embodiments, the reserve power system maybe heavily oversubscribed. For example, in one embodiment, the totalload of the computer systems to which the reserve power system iscoupled is more than 10 times the capacity of the reserve UPS.

At 482, reserve power is supplied to at least one of the sets ofcomputer systems when at least one condition is met. Power may beswitched over automatically, such as by way of the automatic transferswitches described above relative to FIGS. 1-6. The condition may be,for example, an at least partial failure of a primary power system.

At 484, the sets of computer systems are prioritized. At 486, status ofat least one condition in the reserve power system is monitored. Thecondition may be, for example, a predetermined current level in areserve power system. At 488, if no faults are detected, reserve powersystem may remain in full operation. If a fault is detected in one ormore of the sets of computer systems (such as the lowest priority set ofcomputer systems) may be shut down to protect against overload of thereserve power system at 492.

FIG. 8 is a flow diagram illustrating failover logic for a reserve powersystem. The failover logic may be implemented, for example, in aprogrammable logic controller such as described above relative toFIG. 1. At 502, a primary switchboard is powered up. At 504, the primaryswitchboard may be monitored for a failure condition. If no failure hasoccurred upon start up of the primary switchboard, primary UPSs 1, 2, 3may be brought into service at 508 as the primary switchboard continuesto operate. At 510, primary UPSs 1, 2, 3 may be monitored for failureconditions. If no failure has occurred upon start up of primary UPSs 1,2, 3, primary PDUs 1-9 may be brought into service at 514 as the primaryswitchboard and primary UPSs 1,2, 3 continues to operate. At 516,primary PDUs 1-9 may be monitored for failure conditions at 516. If nofailure has occurred upon start up of the primary PDUs 1-9, loads may bebrought online at 532. Loads may be, for example, from servers in a datacenter.

During computing operations, monitoring may continue of the primaryswitchboard, primary UPSs 1, 2, 3, and PDUs 1-9 at 504, 510, and 516,respectively. If a failure condition is detected in the primaryswitchboard, the loads may be switched to reserve switchboard at 522. Ifa failure condition is detected in any or all of primary UPSs 1, 2, 3,the loads may be switched to reserve UPS at 526. If a failure conditionis detected in any or all of primary PDUs 1-9, the loads may be switchedto one of reserve PDUs 1-3 at 530.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

1-28. (canceled)
 29. A system for supplying power, comprising: one ormore sets of computer systems; one or more primary power systems coupledto, and configured to supply primary power to, the one or more sets ofcomputer systems, wherein at least one of the one or more primary powersystems comprises: a downstream portion configured to distribute powerto the one or more sets of computer systems, a mid-stream portionconfigured to supply received power to the downstream portion, and anupstream portion configured to selectively supply primary power from oneor more primary power sources to the mid-stream portion; and a reservepower system coupled to, and configured to automatically supply reservepower to, the one or more sets of computer systems, wherein the totalpower requirements of the one or more sets of computer systems exceedthe capacity of the reserve power system to supply power; wherein thereserve power system is configured to detect a failure in the upstreamportion of the at least one primary power system; and wherein, inresponse to detecting the failure, the reserve power system isconfigured to selectively switch the mid-stream portion from supplyingprimary power received from the upstream portion of the primary powersystem to supplying reserve power received from an upstream portion ofthe reserve power system.
 30. The system of claim 29, wherein: theupstream portion of the reserve power system further comprises one ormore automatic transfer switches; and wherein the one or more automatictransfer switches are configured to automatically switch the mid-streamportion of the at least one primary power system from supplying primarypower received from the upstream portion of the at least one primarypower system to supplying reserve power received from the upstreamportion of the reserve power system.
 31. The system of claim 29,wherein: the one or more sets of computer systems are included in one ormore computing modules that comprise a first computing module and asecond computing module; the one or more primary power systems comprisea first primary power system configured to supply power to the firstcomputing module and a second primary power system configured to supplyprimary power to the second computing module; and the reserve powersystem is configured to: automatically supply power to the firstcomputing module via a mid-stream portion of the first primary powersystem if one or more components in an upstream portion of the firstprimary power system fail, and automatically supply power to the secondcomputing module via a mid-stream portion of the second primary powersystem if one or more components in an upstream portion of the secondprimary power system fail.
 32. The system of claim 29, wherein: themid-stream portion of the at least one primary power system comprises aprimary power-side uninterruptible power supply (UPS); and wherein, toselectively switch the mid-stream portion from supplying primary powerreceived to supplying reserve power, the reserve power system isconfigured to selectively switch the primary power-side UPS fromreceiving primary power to receiving reserve power.
 33. The system ofclaim 32, wherein the downstream portion of the at least one primarypower system comprises a primary power-side power distribution unit(PDU) configured to distribute power received from the primarypower-side UPS to the one or more sets of computer systems.
 34. Thesystem of claim 32, wherein: the upstream portion of the at least oneprimary power system comprises a switchgear device configured toselectively route primary power from one of a primary power-sidetransformer or a primary power-side generator to the primary power-sideUPS; and the source of the failure is located in the switchgear device.35. The system of claim 29, wherein: the reserve power system comprisesone or more shut-down devices; and to protect against overload of thereserve power system, at least one of the shut-down devices isconfigured to automatically shut down at least one of the sets ofcomputer systems if at least one condition is met.
 36. The system ofclaim 35, wherein the reserve power system is configured to prioritizeat least one of the computer systems or at least one of the sets ofcomputer systems, wherein the reserve power system is configured toshut-down at least one relatively low priority computer system whilecontinuing to supply reserve power to at least one other computersystem.
 37. The system of claim 29, wherein the at least one of the setsof computer systems corresponds to a rack system in a data center.
 38. Asystem for providing reserve power to computer systems in a data center,comprising: a reserve power system coupled to, and configured toautomatically supply power to one or more sets of computer systemsreceiving power from one or more primary power systems downstream of aprimary power-side switchgear device if at least one condition is met,the reserve power system comprising: a reserve power-side switchgeardevice configured to supply power from one or more reserve power sourcesseparate from one or more primary power sources supplying power to theprimary power-side switchgear device; and an automatic transfer switchconfigured to automatically switch the one or more sets of computersystems from receiving power from a primary power source through theprimary power-side switchgear device and at least one of the primarypower systems to receiving power from the reserve power source throughthe reserve power-side switchgear device and the at least one primarypower system when at least one condition is met; wherein the total powerrequirements of the one or more sets of computer systems coupled to thereserve power system exceed a capacity of the reserve power system tosupply power.
 39. The system of claim 38, wherein the automatic transferswitch comprises: a primary power input coupled to the primarypower-side switchgear device; a reserve power input coupled to thereserve power-side switchgear device; and a power output coupled to aninput of the at least one primary power system; wherein the automatictransfer switch is configured to selectively route power to the at leastone primary power system from one of the primary power-side switchgeardevice or reserve power-side switchgear device.
 40. The system of claim39, wherein: the at least one primary power system comprises a primarypower-side uninterruptible power supply (UPS); and wherein, toselectively route power to the at least one primary power system fromone of the primary power-side switchgear device or reserve power-sideswitchgear device, the automatic transfer switch is configured toselectively route power to a power input of the primary power-side UPS.41. The system of claim 38, wherein: the reserve power system comprisesone or more shut-down devices; and to protect against overload of thereserve power system, at least one of the shut-down devices isconfigured to automatically shut down at least one of the sets ofcomputer systems if at least one condition is met.
 42. The system ofclaim 41, wherein the reserve power system is configured to prioritizeat least one of the computer systems or at least one of the sets ofcomputer systems, wherein the reserve power system is configured toshut-down at least one relatively low priority computer system whilecontinuing to supply reserve power to at least one other computersystem.
 43. The system of claim 38, wherein the automatic transferswitch comprises a manual override.
 44. A method of supplying reservepower to computer systems in a data center, comprising: monitoring aprimary power feed supplied to at least one set of computer systemsthrough a primary power-side switchgear device and a primary powersystem, wherein the primary power system is downstream of the primarypower-side switchgear device; and in response to an at least partialfailure of the primary power feed upstream of the primary power system,automatically switching the primary power system from supplying theprimary power feed to the at least one set of computer systems tosupplying a reserve power feed from a reserve power system to the atleast one set of computer systems, such that the switchgear device isbypassed by the reserve power system; wherein the total powerrequirements of the one or more sets of computer systems exceed thecapacity of the reserve power system to supply power.
 45. The method ofclaim 44, wherein automatically switching the primary power system tosupplying the reserve power feed comprises automatically switching apower source for the primary power system from the primary power-sideswitchgear device to a reserve power-side switchgear device.
 46. Themethod of claim 44, wherein the at least partial failure of the primarypower feed upstream of the primary power system comprises a failure inthe primary power-side switchgear device.
 47. The method of claim 44,further comprising automatically shutting down at least one of thecomputer systems if at least one condition is met to protect againstoverload of the reserve power system.
 48. The method of claim 47,further comprising: prioritizing at least one of the computer systems orsets of computer systems; and automatically shutting down at least oneof the computer systems if at least one condition is met to protectagainst overload of the reserve power system.